The following content has been provided by the University of Erlangen-Nürnberg.

Welcome to the Tuesday morning session. Today we want to continue with the problem of hand-eye calibration.

The important figure you have to keep in mind for all the upcoming discussions is shown here.

We have our world coordinate system up there. We have the coordinate system of the calibration pattern.

We have here our hand linked to the camera. This is a rigid link. There is no change possible while we are moving the whole system.

We move the hand camera system from here to here. We denote the transformation between the hand and the eye by X.

The transformation between the two camera coordinate systems by A. The transformation between the two hand coordinate systems by B.

What we want to compute is X. Basically we have one core equation. That is X times B is A times X.

This is the commuting diagram here that you might know from linear algebra commuting diagrams.

That just means there are two different ways to map one point from here to here. These two mappings should be identical.

The unknown transform that we want to estimate is here the X. The question is how can we compute the X.

The first problem we have considered is how are the points of the calibration pattern that are given in terms of the calibration coordinate system mapped to our camera coordinate system.

How can we estimate this to get this mapping from here to here, this mapping from here to here, and then we can compute A out of that.

The problem is, where is your mobile phone? The problem is I capture an image from here and I capture an image from there.

I want to compute the projection mapping and I want to compute the rotation and translation of the camera.

This will give me A1 and A2. We have a very good understanding how this can be done.

The winter semester students, they remember how we calibrated the SeaArm system where we had a three dimensional calibration pattern.

Today we consider a planar calibration pattern with the property that all the points that we have on the calibration pattern are forming a plane.

If they form a plane, we know that the range coordinate, the depth coordinate can be set to zero.

The implication was, and we have seen this yesterday, that if we multiply the three by four projection matrix with the vector, the homogeneous vector representing our 3D points in terms of homogeneous coordinates in 4D of the calibration pattern,

the third component is zero because we have planar points. The implication of that is that the whole calibration process reduces the three by four matrix to a three by three matrix.

We can reproduce out of the three by three matrix the missing column. Now let's step one step back and let's look at the whole system of equations.

Let's think about this, how engineers should think about these things.

Let's look at this. We have a three by three matrix. How do we call the three by three matrix?

Pardon me? H. And we have P. And this was three by three and the P matrix was three by four.

And if I say I want to characterize the mapping from 3D to 2D using this projection matrix or that one, I can compute the whole mapping by using either H or P.

And as an engineer you have to say, hold on, there is something wrong with it. I mean the projection matrix has 12 components and this matrix up there has nine components.

How can you say these two are equivalent?

You know what happens if the cell phone is ringing here? You have to sing a song. It's up to you.

Was it your girlfriend? Let me call her back.

Let's spend the two minutes for this joke. No? You want to keep her? Okay.

You destroyed my structure of the lecture. Did you notice? Now I'm completely lost. H and P. H was what?

The projection matrix with the lost third column. So once again if I multiply, and that's what you should remember, if I multiply P, which is given by P1, P2, P3, P4, these are the columns of P.

If you multiply this with X, Y, 0 and 1, I think we call it H, X, H, I don't know, it doesn't matter how you call things.

Then basically you see that this here is weighted by 0 and that basically means this can disappear and that means H is P1, P2, P4.

And we also have seen that the rotation matrix can be written in terms of a product of three matrices, that's the K matrix, the projection itself, and the rotation translation part.

And then we can combine the rotation and translation part with the projection part and the things break down to the following product. You have the K matrix times

the K matrix times what? Times R times T. I don't know why my pen is so fat here.

I zoomed in at the moment. Hey! Oh, that's nice, I can write large letters. It looks much better, right?

Thank you for the hint. And this is a 3x4 matrix, right? That's what we have seen yesterday, 3x4, and this here is a 3x3 matrix.

And the matrix H is basically given by K, R1, R2, and T.

And then I told you that there are a few properties and we need to know about rotation matrices. And you have to remember these.

Some of them. R is orthonormal. It's an orthonormal matrix. That means the column vectors are unit vectors and they are mutually orthogonal to each other.

And that basically tells us if R is a 3x3 rotation matrix and given R1, R2, we can compute R3 by the cross product.

So it's R3 is R1 cross R2.

That's the third vector. It has to be orthogonal and has to have unit lengths. Why does R3 have unit lengths?

Because R is orthonormal, so all vectors have length 1.

Yeah, but we say R1 and R2 are unit vectors and orthogonal to each other. We compute a third one. Why is this a unit length vector?

Why does the cross product lead to a unit vector?

It's one property of the cross product. If we multiply two unit vectors, the result is also a unit vector.